Faces in Event Streams (FES): An Annotated Face Dataset for Event Cameras.

The use of event-based cameras in computer vision is a growing research direction. However, despite the existing research on face detection using the event camera, a substantial gap persists in the availability of a large dataset featuring annotations for faces and facial landmarks on event streams, thus hampering the development of applications in this direction. In this work, we address this issue by publishing the first large and varied dataset (Faces in Event Streams) with a duration of 689 min for face and facial landmark detection in direct event-based camera outputs. In addition, this article presents 12 models trained on our dataset to predict bounding box and facial landmark coordinates with an mAP50 score of more than 90%. We also performed a demonstration of real-time detection with an event-based camera using our models.

Image Captioning for the Visually Impaired and Blind: A Recipe for Low-Resource Languages.

Visually impaired and blind people often face a range of socioeconomic problems that can make it difficult for them to live independently and participate fully in society. Advances in machine learning pave new venues to implement assistive devices for the visually impaired and blind. In this work, we combined image captioning and text-to-speech technologies to create an assistive device for the visually impaired and blind. Our system can provide the user with descriptive auditory feedback in the Kazakh language on a scene acquired in real-time by a head-mounted camera. The image captioning model for the Kazakh language provided satisfactory results in both quantitative metrics and subjective evaluation. Finally, experiments with a visually unimpaired blindfolded participant demonstrated the feasibility of our approach.

A Central Asian Food Dataset for Personalized Dietary Interventions.

Nowadays, it is common for people to take photographs of every beverage, snack, or meal they eat and then post these photographs on social media platforms. Leveraging these social trends, real-time food recognition and reliable classification of these captured food images can potentially help replace some of the tedious recording and coding of food diaries to enable personalized dietary interventions. Although Central Asian cuisine is culturally and historically distinct, there has been little published data on the food and dietary habits of people in this region. To fill this gap, we aim to create a reliable dataset of regional foods that is easily accessible to both public consumers and researchers. To the best of our knowledge, this is the first work on the creation of a Central Asian Food Dataset (CAFD). The final dataset contains 42 food categories and over 16,000 images of national dishes unique to this region. We achieved a classification accuracy of 88.70% (42 classes) on the CAFD using the ResNet152 neural network model. The food recognition models trained on the CAFD demonstrate the effectiveness and high accuracy of computer vision for dietary assessment.

COVID-19 Vaccination Strategies Considering Hesitancy Using Particle-Based Epidemic Simulation.

Vaccine hesitancy is one of the critical factors in achieving herd immunity and suppressing the COVID-19 epidemic. Many countries face this as an acute public health issue that diminishes the efficacy of their vaccination campaigns. Epidemic modeling and simulation can be used to predict the effects of different vaccination strategies. In this work, we present an open-source particle-based COVID-19 simulator with a vaccination module capable of taking into account the vaccine hesitancy of the population. To demonstrate the efficacy of the simulator, we conducted extensive simulations for the province of Lecco, Italy. The results indicate that the combination of both high vaccination rate and low hesitancy leads to faster epidemic suppression.

Input Agnostic Deep Learning for Alzheimer's Disease Classification Using Multimodal MRI Images.

Alzheimer's disease (AD) is a progressive brain disorder that causes memory and functional impairments. The advances in machine learning and publicly available medical datasets initiated multiple studies in AD diagnosis. In this work, we utilize a multi-modal deep learning approach in classifying normal cognition, mild cognitive impairment and AD classes on the basis of structural MRI and diffusion tensor imaging (DTI) scans from the OASIS-3 dataset. In addition to a conventional multi-modal network, we also present an input agnostic architecture that allows diagnosis with either sMRI or DTI scan, which distinguishes our method from previous multi-modal machine learning-based methods. The results show that the input agnostic model achieves 0.96 accuracy when both structural MRI and DTI scans are provided as inputs.

A Particle-Based COVID-19 Simulator With Contact Tracing and Testing.

Goal: The COVID-19 pandemic has emerged as the most severe public health crisis in over a century. As of January 2021, there are more than 100 million cases and 2.1 million deaths. For informed decision making, reliable statistical data and capable simulation tools are needed. Our goal is to develop an epidemic simulator that can model the effects of random population testing and contact tracing. Methods: Our simulator models individuals as particles with the position, velocity, and epidemic status states on a 2D map and runs an SEIR epidemic model with contact tracing and testing modules. The simulator is available on GitHub under the MIT license. Results: The results show that the synergistic use of contact tracing and massive testing is effective in suppressing the epidemic (the number of deaths was reduced by 72%). Conclusions: The Particle-based COVID-19 simulator enables the modeling of intervention measures, random testing, and contact tracing, for epidemic mitigation and suppression.

A Vaccination Simulator for COVID-19: Effective and Sterilizing Immunization Cases.

In this work, we present a particle-based SEIR epidemic simulator as a tool to assess the impact of different vaccination strategies on viral propagation and to model sterilizing and effective immunization outcomes. The simulator includes modules to support contact tracing of the interactions amongst individuals and epidemiological testing of the general population. The particles are distinguished by age to represent more accurately the infection and mortality rates. The tool can be calibrated by region of interest and for different vaccination strategies to enable locality-sensitive virus mitigation policy measures and resource allocation. Moreover, the vaccination policy can be simulated based on the prioritization of certain age groups or randomly vaccinating individuals across all age groups. The results based on the experience of the province of Lecco, Italy, indicate that the simulator can evaluate vaccination strategies in a way that incorporates local circumstances of viral propagation and demographic susceptibilities. Further, the simulator accounts for modeling the distinction between sterilizing immunization, where immunized people are no longer contagious, and effective immunization, where the individuals can transmit the virus even after getting immunized. The parametric simulation results showed that the sterilizing-age-based vaccination scenario results in the least number of deaths. Furthermore, it revealed that older people should be vaccinated first to decrease the overall mortality rate. Also, the results showed that as the vaccination rate increases, the mortality rate between the scenarios shrinks.

SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams.

We present SpeakingFaces as a publicly-available large-scale multimodal dataset developed to support machine learning research in contexts that utilize a combination of thermal, visual, and audio data streams; examples include human-computer interaction, biometric authentication, recognition systems, domain transfer, and speech recognition. SpeakingFaces is comprised of aligned high-resolution thermal and visual spectra image streams of fully-framed faces synchronized with audio recordings of each subject speaking approximately 100 imperative phrases. Data were collected from 142 subjects, yielding over 13,000 instances of synchronized data (∼3.8 TB). For technical validation, we demonstrate two baseline examples. The first baseline shows classification by gender, utilizing different combinations of the three data streams in both clean and noisy environments. The second example consists of thermal-to-visual facial image translation, as an instance of domain transfer.

End-to-End Deep Diagnosis of X-ray Images.

We present an end-to-end deep learning frame-work for X-ray image diagnosis. As the first step, our system determines whether a submitted image is an X-ray or not. After it classifies the type of the X-ray, it runs the dedicated abnormality classification network. In this work, we only focus on the chest X-rays for abnormality classification. However, the system can be extended to other X-ray types easily. Our deep learning classifiers are based on DenseNet-121 architecture. The test set accuracy obtained for 'X-ray or Not', 'X-ray Type Classification', and 'Chest Abnormality Classification' tasks are 0.987, 0.976, and 0.947, respectively, resulting into an end-to-end accuracy of 0.91. For achieving better results than the state-of-the-art in the 'Chest Abnormality Classification', we utilize the new RAdam optimizer. We also use Gradient-weighted Class Activation Mapping for visual explanation of the results. Our results show the feasibility of a generalized online projectional radiography diagnosis system.

Deep Vibro-Tactile Perception for Simultaneous Texture Identification, Slip Detection, and Speed Estimation.

Autonomous dexterous manipulation relies on the ability to recognize an object and detect its slippage. Dynamic tactile signals are important for object recognition and slip detection. An object can be identified based on the acquired signals generated at contact points during tactile interaction. The use of vibrotactile sensors can increase the accuracy of texture recognition and preempt the slippage of a grasped object. In this work, we present a Deep Learning (DL) based method for the simultaneous texture recognition and slip detection. The method detects non-slip and slip events, the velocity, and discriminate textures-all within 17 ms. We evaluate the method for three objects grasped using an industrial gripper with accelerometers installed on its fingertips. A comparative analysis of convolutional neural networks (CNNs), feed-forward neural networks, and long short-term memory networks confirmed that deep CNNs have a higher generalization accuracy. We also evaluated the performance of the highest accuracy method for different signal bandwidths, which showed that a bandwidth of 125 Hz is enough to classify textures with 80% accuracy.

Erratum: Human grasping database for activities of daily living with depth, color and kinematic data streams.

This corrects the article DOI: 10.1038/sdata.2018.101.

User-Independent Intent Recognition for Lower Limb Prostheses Using Depth Sensing.

OBJECTIVE: The intent recognizers of advanced lower limb prostheses utilize mechanical sensors on the prosthesis and/or electromyographic measurements from the residual limb. Besides the delay caused by these signals, such systems require user-specific databases to train the recognizers. In this paper, our objective is the development and validation of a user-independent intent recognition framework utilizing depth sensing. METHODS: We collected a depth image dataset from 12 healthy subjects engaging in a variety of routine activities. After filtering the depth images, we extracted simple features employing a recursive strategy. The feature vectors were classified using a support vector machine. For robust activity mode switching, we implemented a voting filter scheme. RESULTS: The model selection showed that the support vector machine classifier with no dimension reduction has the highest classification accuracy. Specifically, it reached 94.1% accuracy on the testing data from four subjects. We also observed a positive trend in the accuracy of classifiers trained with data from increasing the number of subjects. Activity mode switching using a voting filter detected 732 out of 778 activity mode transitions of the four users while initiating 70 erroneous transitions during steady-state activities. CONCLUSION: The intent recognizer trained on multiple subjects can be used for any other subject, providing a promising solution for supervisory control of powered lower limb prostheses. SIGNIFICANCE: A user-independent intent recognition framework has the potential to decrease or eliminate the time required for extensive data collection regiments for intent recognizer training. This could accelerate the introduction of robotic lower limb prostheses to the market.

Human grasping database for activities of daily living with depth, color and kinematic data streams.

This paper presents a grasping database collected from multiple human subjects for activities of daily living in unstructured environments. The main strength of this database is the use of three different sensing modalities: color images from a head-mounted action camera, distance data from a depth sensor on the dominant arm and upper body kinematic data acquired from an inertial motion capture suit. 3826 grasps were identified in the data collected during 9-hours of experiments. The grasps were grouped according to a hierarchical taxonomy into 35 different grasp types. The database contains information related to each grasp and associated sensor data acquired from the three sensor modalities. We also provide our data annotation software written in Matlab as an open-source tool. The size of the database is 172 GB. We believe this database can be used as a stepping stone to develop big data and machine learning techniques for grasping and manipulation with potential applications in rehabilitation robotics and intelligent automation.

A feasibility study of depth image based intent recognition for lower limb prostheses.

This paper presents our preliminary work on a depth camera based intent recognition system intended for future use in robotic prosthetic legs. The approach infers the activity mode of the subject for standing, walking, running, stair ascent and stair descent modes only using data from the depth camera. Depth difference images are also used to increase the performance of the approach by discriminating between static and dynamic instances. After confidence map based filtering, simple features such as mean, maximum, minimum and standard deviation are extracted from rectangular regions of the frames. A support vector machine with a cubic kernel is used for the classification task. The classification results are post-processed by a voting filter to increase the robustness of activity mode recognition. Experiments conducted with a healthy subject donning the depth camera to his lower leg showed the efficacy of the approach. Specifically, the depth camera based recognition system was able to identify 28 activity mode transitions successfully. The only case of incorrect mode switching was an intended run to stand transition, where an intermediate transition from run to walk was recognized before transitioning to the intended standing mode.

Vital sign monitoring utilizing Eulerian video magnification and thermography.

In this paper we present a proof of concept for non-contact extraction of vital signs using RGB and thermal images obtained from a smart phone. Using our method, heart rate, respiratory rate and forehead temperature can be measured concurrently. Face detection and tracking is leveraged in order to allow natural motion of patients. Heart rate is estimated via processing of visible band RGB video using Eulerian Video Magnification technique. Respiratory rate and the temperature is measured using thermal video. Experiments conducted with 11 healthy subjects indicate that heart rate and respiration rate can be measured with 92 and 94 percent accuracy, respectively.

MOSES: A Matlab-based open-source stochastic epidemic simulator.

This paper presents an open-source stochastic epidemic simulator. Discrete Time Markov Chain based simulator is implemented in Matlab. The simulator capable of simulating SEQIJR (susceptible, exposed, quarantined, infected, isolated and recovered) model can be reduced to simpler models by setting some of the parameters (transition probabilities) to zero. Similarly, it can be extended to more complicated models by editing the source code. It is designed to be used for testing different control algorithms to contain epidemics. The simulator is also designed to be compatible with a network based epidemic simulator and can be used in the network based scheme for the simulation of a node. Simulations show the capability of reproducing different epidemic model behaviors successfully in a computationally efficient manner.

Medical decision support using machine learning for early detection of late-onset neonatal sepsis.

OBJECTIVE: The objective was to develop non-invasive predictive models for late-onset neonatal sepsis from off-the-shelf medical data and electronic medical records (EMR). DESIGN: The data used in this study are from 299 infants admitted to the neonatal intensive care unit in the Monroe Carell Jr. Children's Hospital at Vanderbilt and evaluated for late-onset sepsis. Gold standard diagnostic labels (sepsis negative, culture positive sepsis, culture negative/clinical sepsis) were assigned based on all the laboratory, clinical and microbiology data available in EMR. Only data that were available up to 12 h after phlebotomy for blood culture testing were used to build predictive models using machine learning (ML) algorithms. MEASUREMENT: We compared sensitivity, specificity, positive predictive value and negative predictive value of sepsis treatment of physicians with the predictions of models generated by ML algorithms. RESULTS: The treatment sensitivity of all the nine ML algorithms and specificity of eight out of the nine ML algorithms tested exceeded that of the physician when culture-negative sepsis was included. When culture-negative sepsis was excluded both sensitivity and specificity exceeded that of the physician for all the ML algorithms. The top three predictive variables were the hematocrit or packed cell volume, chorioamnionitis and respiratory rate. CONCLUSIONS: Predictive models developed from off-the-shelf and EMR data using ML algorithms exceeded the treatment sensitivity and treatment specificity of clinicians. A prospective study is warranted to assess the clinical utility of the ML algorithms in improving the accuracy of antibiotic use in the management of neonatal sepsis.

Control of stair ascent and descent with a powered transfemoral prosthesis.

This paper presents a finite state-based control system for a powered transfemoral prosthesis that provides stair ascent and descent capability. The control system was implemented on a powered prosthesis and evaluated by a unilateral, transfemoral amputee subject. The ability of the powered prosthesis to provide stair ascent and descent capability was assessed by comparing the gait kinematics, as recorded by a motion capture system, with the kinematics provided by a passive prosthesis, in addition to those recorded from a set of healthy subjects. The results indicate that the powered prosthesis provides gait kinematics that are considerably more representative of healthy gait, relative to the passive prosthesis, for both stair ascent and descent.

A method for the control of multigrasp myoelectric prosthetic hands.

This paper presents the design and preliminary experimental validation of a multigrasp myoelectric controller. The described method enables direct and proportional control of multigrasp prosthetic hand motion among nine characteristic postures using two surface electromyography electrodes. To assess the efficacy of the control method, five nonamputee subjects utilized the multigrasp myoelectric controller to command the motion of a virtual prosthesis between random sequences of target hand postures in a series of experimental trials. For comparison, the same subjects also utilized a data glove, worn on their native hand, to command the motion of the virtual prosthesis for similar sequences of target postures during each trial. The time required to transition from posture to posture and the percentage of correctly completed transitions were evaluated to characterize the ability to control the virtual prosthesis using each method. The average overall transition times across all subjects were found to be 1.49 and 0.81 s for the multigrasp myoelectric controller and the native hand, respectively. The average transition completion rates for both were found to be the same (99.2%). Supplemental videos demonstrate the virtual prosthesis experiments, as well as a preliminary hardware implementation.

Standing stability enhancement with an intelligent powered transfemoral prosthesis.

The authors have developed a ground-adaptive standing controller for a powered knee and ankle prosthesis which is intended to enhance the standing stability of transfemoral amputees. The finite-state-based controller includes a ground-searching phase, a slope estimation phase, and a joint impedance modulation phase, which together enable the prosthesis to quickly conform to the ground and provide stabilizing assistance to the user. In order to assess the efficacy of the ground-adaptive standing controller, the control approach was implemented on a powered knee and ankle prosthesis, and experimental data were collected on an amputee subject for a variety of standing conditions. Results indicate that the controller can estimate the ground slope within ±1° over a range of ±15°, and that it can provide appropriate joint impedances for standing on slopes within this range.